{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "068f6458-0575-4905-b6a0-7fa511b9b621",
   "metadata": {},
   "source": [
    "### 构建一个语义搜索引擎\n",
    "\n",
    "本教程将帮助你熟悉 LangChain 的几个核心组件：**文档加载器（document loader）**、**嵌入模型（embedding）** 和 **向量存储（vector store）** 的抽象接口。这些抽象的设计目的是为了支持从（向量）数据库和其他数据源中检索数据，以便将其集成到大型语言模型（LLM）的工作流程中。\n",
    "\n",
    "这些功能对于那些在模型推理过程中需要结合外部数据进行思考的应用非常重要，例如在 **检索增强生成（Retrieval-Augmented Generation，RAG）** 中的应用。\n",
    "\n",
    "在本教程中，我们将构建一个针对 PDF 文档的搜索引擎。这将使我们能够根据输入的查询语句，检索出 PDF 中与之相似的段落内容。\n",
    "\n",
    "---\n",
    "\n",
    "### 涉及的核心概念\n",
    "\n",
    "本指南主要介绍文本数据的检索方法，涵盖以下核心概念：\n",
    "\n",
    "1. **文档与文档加载器（Documents and document loaders）**  \n",
    "   如何加载和表示文档数据。\n",
    "\n",
    "2. **文本分割器（Text splitters）**  \n",
    "   将大段文本切分为适合处理的小块内容。\n",
    "\n",
    "3. **嵌入模型（Embeddings）**  \n",
    "   将文本转换为可用于语义匹配的向量表示。\n",
    "\n",
    "4. **向量存储与检索器（Vector stores and retrievers）**  \n",
    "   存储和高效检索向量化后的文本片段。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bcc0871-0cdc-4074-9248-e20d68f46b3c",
   "metadata": {},
   "source": [
    "### 安装\n",
    "\n",
    "本教程需要安装 `langchain-community` 和 `pypdf` 两个 Python 包："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aff1c984-725b-41f6-bf89-8e17601af22a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --upgrade --quiet langchain-community pypdf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f1cd28b-5d64-4c17-a60a-c5aeb23661cc",
   "metadata": {},
   "source": [
    "### 文档和文档加载器\n",
    "LangChain 实现了一个 **Document（文档）** 抽象类，用于表示一段文本内容及其相关的元数据。它具有三个属性：\n",
    "\n",
    "- `page_content`：一个字符串，表示文档的内容；\n",
    "- `metadata`：一个字典，包含任意的元数据；\n",
    "- `id`：（可选）文档的唯一标识符。\n",
    "\n",
    "其中，`metadata` 属性可以记录文档来源、与其他文档的关系等信息。请注意，一个单独的 `Document` 对象通常代表一个更大文档中的“块”（chunk）。  \n",
    "\n",
    "LangChain 生态系统中实现了许多**文档加载器（document loaders）**，它们与数百种常见的数据源进行了集成。这使得将这些数据源中的内容整合到你的 AI 应用程序中变得非常简便。\n",
    "\n",
    "在需要时，我们也可以生成示例文档："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "885e3492-dc83-4226-a81c-4a8d85d430b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.documents import Document\n",
    "\n",
    "documents = [\n",
    "    Document(\n",
    "        page_content=\"狗是很好的伙伴，以忠诚和友好而闻名。\",\n",
    "        metadata={\"source\": \"mammal-pets-doc\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"猫是独立的宠物，通常喜欢拥有自己的空间。\",\n",
    "        metadata={\"source\": \"mammal-pets-doc\"},\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "580f4295-4bb9-4561-9be5-2b2846778482",
   "metadata": {},
   "source": [
    "### 加载文档\n",
    "\n",
    "让我们将一个 PDF 文件加载为一系列 `Document` 对象。\n",
    "\n",
    "在 LangChain 仓库中有一个示例 PDF 文件 —— 这是耐克公司（Nike）2023 年的 10-K 财报文件。\n",
    "\n",
    "我们可以参考 [LangChain 文档](https://python.langchain.com/docs/integrations/document_loaders/#pdfs) 中关于支持的 PDF 文档加载器的说明。我们选择使用 `PyPDFLoader`，这是一个相对较轻量级的加载器。\n",
    "\n",
    "然后你就可以使用 `PyPDFLoader` 来加载 PDF 文件了。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4fb31506-5098-4027-a414-97ec1c94baea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "107\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "file_path = \"../assets/example_data/nke-10k-2023.pdf\"\n",
    "loader = PyPDFLoader(file_path)\n",
    "\n",
    "docs = loader.load()\n",
    "\n",
    "print(len(docs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f46b7a8-18c9-4024-bee9-d5b8d0c9c5c0",
   "metadata": {},
   "source": [
    "`PyPDFLoader` 会为 PDF 中的每一页加载一个 `Document` 对象。对于每一个对象，我们可以轻松访问：\n",
    "\n",
    "- 页面的字符串内容（`page_content`）；\n",
    "- 包含文件名和页码的元数据（`metadata`）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f55222e6-9f8b-4a45-8fb8-769c0e668d4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Table of Contents\n",
      "UNITED STATES\n",
      "SECURITIES AND EXCHANGE COMMISSION\n",
      "Washington, D.C. 20549\n",
      "FORM 10-K\n",
      "(Mark One)\n",
      "☑  ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\n",
      "F\n",
      "\n",
      "{'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 0, 'page_label': '1'}\n"
     ]
    }
   ],
   "source": [
    "print(f\"{docs[0].page_content[:200]}\\n\")\n",
    "print(docs[0].metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "123ef5cf-1945-40d7-801c-e4f3e0c8503c",
   "metadata": {},
   "source": [
    "### 分割（Splitting）\n",
    "\n",
    "出于信息检索和下游问答的目的，单个 PDF 页面的粒度可能过于粗糙。我们最终的目标是检索能够回答输入查询的 `Document` 对象，因此进一步对文档进行分割将有助于确保文档中相关部分的含义不会被周围文本“淹没”。\n",
    "\n",
    "我们可以使用 **文本分割器（text splitters）** 来实现这一目标。在这里，我们将使用一个基于字符的简单文本分割器。我们将文档分割为每块 1000 个字符，并在块之间保留 200 个字符的重叠部分。这种重叠有助于避免将某个语句与其相关的上下文分隔开。\n",
    "\n",
    "我们使用的是 `RecursiveCharacterTextSplitter`，它会递归地使用常见的分隔符（如换行符）对文档进行分割，直到每个块达到合适的大小。这是推荐用于通用文本场景的文本分割器。\n",
    "\n",
    "我们设置 `add_start_index=True`，这样每个分割后的 `Document` 在原始文档中开始的字符索引会被保存为元数据中的 `\"start_index\"` 属性。\n",
    "\n",
    "---\n",
    "\n",
    "📌 **想了解更多关于如何处理 PDF 的内容**，包括如何提取特定章节的文本和图像，请参阅 [LangChain 关于 PDF 处理的指南](https://python.langchain.com/docs/how_to/document_loader_pdf/)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "082cf5c7-afe1-40c1-9ffe-38780cc8cafe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "516"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=1000, chunk_overlap=200, add_start_index=True\n",
    ")\n",
    "all_splits = text_splitter.split_documents(docs)\n",
    "\n",
    "len(all_splits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "49ce83fd-fab7-4770-9f4b-01b892ec6cd4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 0, 'page_label': '1', 'start_index': 0}, page_content=\"Table of Contents\\nUNITED STATES\\nSECURITIES AND EXCHANGE COMMISSION\\nWashington, D.C. 20549\\nFORM 10-K\\n(Mark One)\\n☑  ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\\nFOR THE FISCAL YEAR ENDED MAY 31, 2023\\nOR\\n☐  TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\\nFOR THE TRANSITION PERIOD FROM                         TO                         .\\nCommission File No. 1-10635\\nNIKE, Inc.\\n(Exact name of Registrant as specified in its charter)\\nOregon 93-0584541\\n(State or other jurisdiction of incorporation) (IRS Employer Identification No.)\\nOne Bowerman Drive, Beaverton, Oregon 97005-6453\\n(Address of principal executive offices and zip code)\\n(503) 671-6453\\n(Registrant's telephone number, including area code)\\nSECURITIES REGISTERED PURSUANT TO SECTION 12(B) OF THE ACT:\\nClass B Common Stock NKE New York Stock Exchange\\n(Title of each class) (Trading symbol) (Name of each exchange on which registered)\")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_splits[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0984a46c-4dbd-4191-a8b5-b4d1c52c3927",
   "metadata": {},
   "source": [
    "### 词嵌入\n",
    "**向量搜索** 是一种用于存储和检索非结构化数据（例如非结构化文本）的常用方法。其核心思想是将与文本相关的数值向量进行存储。当给定一个查询时，我们可以将其也嵌入为相同维度的向量，并使用向量相似度度量方法（如余弦相似度）来识别相关的文本内容。\n",
    "\n",
    "LangChain 支持来自多个提供商的嵌入模型（embeddings），这些模型定义了如何将文本转换为数值向量。下面我们选择一个模型："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85f7daee-abb8-47d5-ad63-97395d6e7442",
   "metadata": {},
   "source": [
    "### ✅ 正确搭配建议\n",
    "\n",
    "| 场景 | 推荐 Embedding 模型 | 推荐 LLM |\n",
    "|------|---------------------|-----------|\n",
    "| 文本 Embedding + 语义检索 | `text-embedding-v1/v2` | N/A |\n",
    "| 多模态理解（图文） | N/A | `qwen-vl-max` |\n",
    "| RAG（基于文本检索 + 生成） | `text-embedding-v1/v2` | `qwen-turbo` / `qwen-plus` / `qwen-max` |\n",
    "\n",
    "---\n",
    "\n",
    "### ✅ 示例：正确搭配使用\n",
    "\n",
    "```python\n",
    "# 使用 Qwen 的 Embedding 模型进行向量化\n",
    "from langchain_community.embeddings import TongyiEmbeddings\n",
    "embeddings = TongyiEmbeddings(model=\"text-embedding-v1\")\n",
    "\n",
    "# 使用 Qwen 的 LLM 进行回答生成\n",
    "from langchain_community.chat_models import ChatTongyi\n",
    "chat_model = ChatTongyi(model=\"qwen-turbo\")\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "### 📌 总结\n",
    "\n",
    "| 模型名称 | 类型 | 是否可用于 Embedding | 是否适合与 qwen-vl-max 搭配 |\n",
    "|----------|------|------------------------|------------------------------|\n",
    "| `qwen-vl-max` | 多模态 LLM | ❌（不适合） | ❌ 不推荐作为 Embedding 使用 |\n",
    "| `text-embedding-v1/v2` | Embedding 模型 | ✅ | ✅ 推荐用于语义检索等任务 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5fb8eb0b-f7ce-4f76-8352-cacd6e1b61d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install --upgrade --quiet  langchain-community dashvector dashscope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eb13a9f8-1be5-4b83-9beb-5ffbee6bffa5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "try:\n",
    "    # load environment variables from .env file (requires `python-dotenv`)\n",
    "    from dotenv import load_dotenv\n",
    "\n",
    "    _ = load_dotenv()\n",
    "except ImportError:\n",
    "    pass\n",
    "\n",
    "if not os.environ.get(\"DASHSCOPE_API_KEY\"):\n",
    "  os.environ[\"DASHSCOPE_API_KEY\"] = getpass.getpass(\"Enter API key for OpenAI: \")\n",
    "\n",
    "from langchain_community.embeddings.dashscope import DashScopeEmbeddings\n",
    "# 初始化 Qwen Embedding 模型\n",
    "embeddings = DashScopeEmbeddings(model=\"text-embedding-v1\")  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6e6020d1-21d9-411f-878a-670734807026",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated vectors of length 1536\n",
      "\n",
      "[-3.75884747505188, -13.396482467651367, 7.487827777862549, 4.250734806060791, 2.5559475421905518, 2.882380485534668, -5.610165596008301, -0.058700062334537506, 0.01584913395345211, -0.28392794728279114]\n"
     ]
    }
   ],
   "source": [
    "vector_1 = embeddings.embed_query(all_splits[0].page_content)\n",
    "vector_2 = embeddings.embed_query(all_splits[1].page_content)\n",
    "\n",
    "assert len(vector_1) == len(vector_2)\n",
    "print(f\"Generated vectors of length {len(vector_1)}\\n\")\n",
    "print(vector_1[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c039d740-56a3-4c05-ba34-ee71ef4f026a",
   "metadata": {},
   "source": [
    "在拥有了用于生成文本嵌入（text embeddings）的模型之后，接下来我们可以将这些向量存储在一个特殊的数据结构中，该结构支持高效的相似性搜索。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3daf1f4-ead6-45b1-954d-5a3d74a395c4",
   "metadata": {},
   "source": [
    "### 向量存储（Vector Stores）\n",
    "\n",
    "LangChain 中的 `VectorStore` 对象包含用于**添加文本和文档对象**到存储中，以及使用各种**相似性度量方法**进行查询的方法。它们通常在初始化时会绑定一个嵌入模型（embedding model），该模型决定了文本数据如何被转换为数值向量。\n",
    "\n",
    "LangChain 提供了一系列与不同向量存储技术集成的功能：\n",
    "\n",
    "- 有些向量存储是由服务提供商托管的（例如各类云服务商），使用时需要特定的凭证；\n",
    "- 有些（如 Postgres）可以在本地运行，或通过第三方服务部署；\n",
    "- 还有一些可以在内存中运行，适用于轻量级任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0b3be2a3-82ca-40db-aae0-7370a5c6174a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.vectorstores import InMemoryVectorStore\n",
    "\n",
    "vector_store = InMemoryVectorStore(embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369aceb7-dff2-4ccc-a456-6f305ebe5975",
   "metadata": {},
   "source": [
    "实例化我们的向量存储后，我们现在可以索引文档。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "496ccd73-270c-4978-9323-ed7b6500d98d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ids = vector_store.add_documents(documents=all_splits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4a30d8e3-bdc3-4a56-a202-5d00eb9ad43c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['9a432618-ff75-4d61-9075-050817d902ec',\n",
       " '24f25b57-8265-4ddf-affc-036e55ed473d',\n",
       " '4adccd4c-db5c-40f3-9270-04717288ffb4',\n",
       " 'ecfcea94-ae1e-427e-bb1f-f29f134a4f84',\n",
       " '8c0bb7b9-2ada-4b41-b2c7-2d1465b58dd8']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ids[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66985a40-a303-46bf-a070-1e21dd85a8c6",
   "metadata": {},
   "source": [
    "请注意，大多数向量存储（Vector Store）的实现都允许你连接到一个**已有的向量存储**——例如，通过提供客户端对象、索引名称或其他相关信息。具体操作可以参考对应集成的文档以获得更详细的说明。\n",
    "\n",
    "一旦我们实例化了一个包含文档的 `VectorStore`，就可以对其进行查询。`VectorStore` 提供了多种查询方式：\n",
    "\n",
    "- **同步查询 和 异步查询**；\n",
    "- **通过字符串查询 和 通过向量查询**；\n",
    "- **返回相似度分数 和 不返回相似度分数**；\n",
    "- **按相似性排序查询 和 按最大边缘相关性（MMR）查询**（用于在结果中平衡与查询的相关性和结果之间的多样性）。\n",
    "\n",
    "这些查询方法的返回结果通常都包含一个 `Document` 对象的列表。\n",
    "\n",
    "### 使用方式\n",
    "\n",
    "嵌入模型（Embeddings）通常将文本表示为一个“稠密”向量，使得在语义上相近的文本在几何空间中距离较近。这使我们能够仅通过传入一个问题来检索相关信息，而无需事先知道文档中使用的任何特定关键词。\n",
    "\n",
    "根据与字符串查询的相似性返回文档："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e5241044-833b-4c34-8b75-32fd9950cc32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='operations. We also lease an office complex in Shanghai, China, our headquarters for our Greater China geography, occupied by employees focused on implementing our\n",
      "wholesale, NIKE Direct and merchandising strategies in the region, among other functions.\n",
      "In the United States, NIKE has eight significant distribution centers. Five are located in or near Memphis, Tennessee, two of which are owned and three of which are\n",
      "leased. Two other distribution centers, one located in Indianapolis, Indiana and one located in Dayton, Tennessee, are leased and operated by third-party logistics\n",
      "providers. One distribution center for Converse is located in Ontario, California, which is leased. NIKE has a number of distribution facilities outside the United States,\n",
      "some of which are leased and operated by third-party logistics providers. The most significant distribution facilities outside the United States are located in Laakdal,' metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 26, 'page_label': '27', 'start_index': 804}\n"
     ]
    }
   ],
   "source": [
    "results = vector_store.similarity_search(\n",
    "    \"How many distribution centers does Nike have in the US?\"\n",
    ")\n",
    "\n",
    "print(results[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8950d9ac-a1f2-401c-ad6c-d1ce24b1e80b",
   "metadata": {},
   "source": [
    "**异步查询：**\n",
    "\n",
    "指的是在不阻塞主线程的情况下执行的查询操作。在处理大型语言模型或远程向量数据库时，查询可能会耗时较长，使用异步方式可以提高程序的响应性和效率。\n",
    "\n",
    "在 LangChain 中，某些 `VectorStore` 实现支持异步方法（如 `asimilarity_search()`），你可以在异步函数中调用它们，通常与 `asyncio` 一起使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4cafe177-017f-4bb9-ba59-6e37737c200b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Table of Contents\n",
      "PART I\n",
      "ITEM 1. BUSINESS\n",
      "GENERAL\n",
      "NIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this \"Annual Report\"), the terms \"we,\" \"us,\" \"our,\"\n",
      "\"NIKE\" and the \"Company\" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\n",
      "Our principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\n",
      "the largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\n",
      "and sales through our digital platforms (also referred to as \"NIKE Brand Digital\"), to retail accounts and to a mix of independent distributors, licensees and sales' metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 3, 'page_label': '4', 'start_index': 0}\n"
     ]
    }
   ],
   "source": [
    "results = await vector_store.asimilarity_search(\"When was Nike incorporated?\") # 耐克是什么时候成立的\n",
    " \n",
    "print(results[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6613bb64-17dc-4c34-836d-0e649b80c466",
   "metadata": {},
   "source": [
    "**返回分数：**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e59e3674-a0c3-4b99-8a6b-a0fa89488fe4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.7932733865566657\n",
      "\n",
      "page_content='Enterprise Resource Planning Platform, data and analytics, demand sensing, insight gathering, and other areas to create an end-to-end technology foundation, which we\n",
      "believe will further accelerate our digital transformation. We believe this unified approach will accelerate growth and unlock more efficiency for our business, while driving\n",
      "speed and responsiveness as we serve consumers globally.\n",
      "FINANCIAL HIGHLIGHTS\n",
      "• In fiscal 2023, NIKE, Inc. achieved record Revenues of $51.2 billion, which increased 10% and 16% on a reported and currency-neutral basis, respectively\n",
      "• NIKE Direct revenues grew 14% from $18.7 billion in fiscal 2022 to $21.3 billion in fiscal 2023, and represented approximately 44% of total NIKE Brand revenues for\n",
      "fiscal 2023\n",
      "• Gross margin for the fiscal year decreased 250 basis points to 43.5% primarily driven by higher product costs, higher markdowns and unfavorable changes in foreign\n",
      "currency exchange rates, partially offset by strategic pricing actions' metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 30, 'page_label': '31', 'start_index': 1540}\n"
     ]
    }
   ],
   "source": [
    "# Note that providers implement different scores; the score here\n",
    "# is a distance metric that varies inversely with similarity.\n",
    "\n",
    "results = vector_store.similarity_search_with_score(\"What was Nike's revenue in 2023?\")  # 耐克 2023 年的收入是多少？\n",
    "doc, score = results[0]\n",
    "print(f\"Score: {score}\\n\")\n",
    "print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43149ae4-9b4b-4425-b909-68e9cce98114",
   "metadata": {},
   "source": [
    "根据嵌入向量的相似性返回文档：\n",
    "\n",
    "指通过将查询文本转换为向量（即嵌入），然后在向量数据库中查找与该向量最相似的文档。这种方式不依赖关键词匹配，而是基于语义相似性进行检索。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5cd1828d-62dd-4956-9c36-7a11cfab5dd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Enterprise Resource Planning Platform, data and analytics, demand sensing, insight gathering, and other areas to create an end-to-end technology foundation, which we\n",
      "believe will further accelerate our digital transformation. We believe this unified approach will accelerate growth and unlock more efficiency for our business, while driving\n",
      "speed and responsiveness as we serve consumers globally.\n",
      "FINANCIAL HIGHLIGHTS\n",
      "• In fiscal 2023, NIKE, Inc. achieved record Revenues of $51.2 billion, which increased 10% and 16% on a reported and currency-neutral basis, respectively\n",
      "• NIKE Direct revenues grew 14% from $18.7 billion in fiscal 2022 to $21.3 billion in fiscal 2023, and represented approximately 44% of total NIKE Brand revenues for\n",
      "fiscal 2023\n",
      "• Gross margin for the fiscal year decreased 250 basis points to 43.5% primarily driven by higher product costs, higher markdowns and unfavorable changes in foreign\n",
      "currency exchange rates, partially offset by strategic pricing actions' metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 30, 'page_label': '31', 'start_index': 1540}\n"
     ]
    }
   ],
   "source": [
    "embedding = embeddings.embed_query(\"How were Nike's margins impacted in 2023?\")\n",
    "\n",
    "results = vector_store.similarity_search_by_vector(embedding)\n",
    "print(results[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5c98a5b5-8550-41d8-abf2-0550d2780230",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(id='48df4447-f488-451d-aba2-0d85a1d4757b', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 30, 'page_label': '31', 'start_index': 1540}, page_content='Enterprise Resource Planning Platform, data and analytics, demand sensing, insight gathering, and other areas to create an end-to-end technology foundation, which we\\nbelieve will further accelerate our digital transformation. We believe this unified approach will accelerate growth and unlock more efficiency for our business, while driving\\nspeed and responsiveness as we serve consumers globally.\\nFINANCIAL HIGHLIGHTS\\n• In fiscal 2023, NIKE, Inc. achieved record Revenues of $51.2 billion, which increased 10% and 16% on a reported and currency-neutral basis, respectively\\n• NIKE Direct revenues grew 14% from $18.7 billion in fiscal 2022 to $21.3 billion in fiscal 2023, and represented approximately 44% of total NIKE Brand revenues for\\nfiscal 2023\\n• Gross margin for the fiscal year decreased 250 basis points to 43.5% primarily driven by higher product costs, higher markdowns and unfavorable changes in foreign\\ncurrency exchange rates, partially offset by strategic pricing actions'),\n",
       " Document(id='bd65e9ac-a811-44d2-a8f9-b034d622f1c8', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 35, 'page_label': '36', 'start_index': 0}, page_content='Table of Contents\\nFISCAL 2023 NIKE BRAND REVENUE HIGHLIGHTSThe following tables present NIKE Brand revenues disaggregated by reportable operating segment, distribution channel and major product line:\\nFISCAL 2023 COMPARED TO FISCAL 2022\\n• NIKE, Inc. Revenues were $51.2 billion in fiscal 2023, which increased 10% and 16% compared to fiscal 2022 on a reported and currency-neutral basis, respectively.\\nThe increase was due to higher revenues in North America, Europe, Middle East & Africa (\"EMEA\"), APLA and Greater China, which contributed approximately 7, 6,\\n2 and 1 percentage points to NIKE, Inc. Revenues, respectively.\\n• NIKE Brand revenues, which represented over 90% of NIKE, Inc. Revenues, increased 10% and 16% on a reported and currency-neutral basis, respectively. This\\nincrease was primarily due to higher revenues in Men\\'s, the Jordan Brand, Women\\'s and Kids\\' which grew 17%, 35%,11% and 10%, respectively, on a wholesale\\nequivalent basis.'),\n",
       " Document(id='80c68722-7408-4066-acfe-e4537653fac8', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 31, 'page_label': '32', 'start_index': 3132}, page_content='of Income. Total NIKE, Inc. EBIT for fiscal 2023 and fiscal 2022 is as follows:\\nYEAR ENDED MAY 31,\\n(Dollars in millions) 2023 2022\\nNet income $ 5,070 $ 6,046\\nAdd: Interest expense (income), net (6) 205\\nAdd: Income tax expense 1,131 605\\nEarnings before interest and taxes $ 6,195 $ 6,856\\nEBIT Margin: Calculated as total NIKE, Inc. EBIT divided by total NIKE, Inc. Revenues. Our EBIT Margin calculation for fiscal 2023 and fiscal 2022 is as follows:\\nYEAR ENDED MAY 31,\\n(Dollars in millions) 2023 2022\\nNumerator\\nEarnings before interest and taxes $ 6,195 $ 6,856\\nDenominator\\nTotal NIKE, Inc. Revenues $ 51,217 $ 46,710\\nEBIT Margin 12.1% 14.7%\\n2023 FORM 10-K 29'),\n",
       " Document(id='a4ae595c-bdcf-40b6-9f05-60b23997cab5', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 42, 'page_label': '43', 'start_index': 666}, page_content='As discussed previously, our NIKE Brand business in Brazil transitioned to a distributor operating model during fiscal 2021. We completed the sale of our entity in Chile\\nand our entities in Argentina and Uruguay to third-party distributors in the first and second quarters of fiscal 2023, respectively. The impacts of closing these transactions\\nare included within Corporate and are not reflected in the APLA operating segment results. This completed the transition of our NIKE Brand businesses within our CASA\\nmarketplace, which now reflects a full distributor operating model. For more information see Note 18 — Acquisitions and Divestitures within the accompanying Notes to\\nthe Consolidated Financial Statements.\\nFISCAL 2023 COMPARED TO FISCAL 2022\\n• APLA revenues increased 17% on a currency-neutral basis due to higher revenues across nearly all territories, led by Southeast Asia and India, Korea and Japan.')]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d68831e1-ee53-4a98-838d-9898ff0515fb",
   "metadata": {},
   "source": [
    "### 检索器（Retrievers）\n",
    "\n",
    "LangChain 中的 `VectorStore` 对象 **没有继承** `Runnable` 接口。而 LangChain 的 **Retrievers（检索器）** 是 `Runnables`，因此它们实现了标准的一组方法（例如：同步和异步的 `invoke` 和 `batch` 操作）。\n",
    "\n",
    "虽然我们可以通过向量存储来构建检索器，但检索器不仅可以与向量存储交互，还可以连接其他类型的数据源（例如外部 API）。\n",
    "\n",
    "我们可以自己创建一个简单的版本，**不需要继承 Retriever 类**，也能实现类似功能。只要我们明确选择使用哪种方法来检索文档，就可以轻松地创建一个可运行的对象（Runnable）。下面我们将基于 `similarity_search` 方法来构建这样一个对象："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "041e8782-70b2-4064-9d6e-0c7ae0d08a53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Document(id='d1325b72-3b04-4371-a846-44cd80e5f06e', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 26, 'page_label': '27', 'start_index': 804}, page_content='operations. We also lease an office complex in Shanghai, China, our headquarters for our Greater China geography, occupied by employees focused on implementing our\\nwholesale, NIKE Direct and merchandising strategies in the region, among other functions.\\nIn the United States, NIKE has eight significant distribution centers. Five are located in or near Memphis, Tennessee, two of which are owned and three of which are\\nleased. Two other distribution centers, one located in Indianapolis, Indiana and one located in Dayton, Tennessee, are leased and operated by third-party logistics\\nproviders. One distribution center for Converse is located in Ontario, California, which is leased. NIKE has a number of distribution facilities outside the United States,\\nsome of which are leased and operated by third-party logistics providers. The most significant distribution facilities outside the United States are located in Laakdal,')],\n",
       " [Document(id='ef0005d7-3d86-482c-898c-0e46b6306487', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 3, 'page_label': '4', 'start_index': 0}, page_content='Table of Contents\\nPART I\\nITEM 1. BUSINESS\\nGENERAL\\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this \"Annual Report\"), the terms \"we,\" \"us,\" \"our,\"\\n\"NIKE\" and the \"Company\" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\\nand sales through our digital platforms (also referred to as \"NIKE Brand Digital\"), to retail accounts and to a mix of independent distributors, licensees and sales')]]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import List\n",
    "\n",
    "from langchain_core.documents import Document\n",
    "from langchain_core.runnables import chain\n",
    "\n",
    "\n",
    "@chain\n",
    "def retriever(query: str) -> List[Document]:\n",
    "    return vector_store.similarity_search(query, k=1)\n",
    "\n",
    "\n",
    "retriever.batch(\n",
    "    [\n",
    "        \"How many distribution centers does Nike have in the US?\",\n",
    "        \"When was Nike incorporated?\",\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78caa0f5-eb34-46bc-9b51-2ca8d38230b3",
   "metadata": {},
   "source": [
    "Vectorstores（向量存储）实现了一个名为 `as_retriever` 的方法，该方法会生成一个 Retriever（检索器），具体来说是一个 `VectorStoreRetriever`（向量存储检索器）。这些检索器包含特定的 `search_type` 和 `search_kwargs` 属性，用于指定调用底层向量存储的哪些方法，以及如何对它们进行参数化。例如，我们可以用以下方式复制上面的效果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6f7759d9-066a-458e-a287-eb11b9dd4653",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Document(id='d1325b72-3b04-4371-a846-44cd80e5f06e', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 26, 'page_label': '27', 'start_index': 804}, page_content='operations. We also lease an office complex in Shanghai, China, our headquarters for our Greater China geography, occupied by employees focused on implementing our\\nwholesale, NIKE Direct and merchandising strategies in the region, among other functions.\\nIn the United States, NIKE has eight significant distribution centers. Five are located in or near Memphis, Tennessee, two of which are owned and three of which are\\nleased. Two other distribution centers, one located in Indianapolis, Indiana and one located in Dayton, Tennessee, are leased and operated by third-party logistics\\nproviders. One distribution center for Converse is located in Ontario, California, which is leased. NIKE has a number of distribution facilities outside the United States,\\nsome of which are leased and operated by third-party logistics providers. The most significant distribution facilities outside the United States are located in Laakdal,')],\n",
       " [Document(id='ef0005d7-3d86-482c-898c-0e46b6306487', metadata={'producer': 'EDGRpdf Service w/ EO.Pdf 22.0.40.0', 'creator': 'EDGAR Filing HTML Converter', 'creationdate': '2023-07-20T16:22:00-04:00', 'title': '0000320187-23-000039', 'author': 'EDGAR Online, a division of Donnelley Financial Solutions', 'subject': 'Form 10-K filed on 2023-07-20 for the period ending 2023-05-31', 'keywords': '0000320187-23-000039; ; 10-K', 'moddate': '2023-07-20T16:22:08-04:00', 'source': '../assets/example_data/nke-10k-2023.pdf', 'total_pages': 107, 'page': 3, 'page_label': '4', 'start_index': 0}, page_content='Table of Contents\\nPART I\\nITEM 1. BUSINESS\\nGENERAL\\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this \"Annual Report\"), the terms \"we,\" \"us,\" \"our,\"\\n\"NIKE\" and the \"Company\" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\\nand sales through our digital platforms (also referred to as \"NIKE Brand Digital\"), to retail accounts and to a mix of independent distributors, licensees and sales')]]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever = vector_store.as_retriever(\n",
    "    search_type=\"similarity\",\n",
    "    search_kwargs={\"k\": 1},\n",
    ")\n",
    "\n",
    "retriever.batch(\n",
    "    [\n",
    "        \"How many distribution centers does Nike have in the US?\",\n",
    "        \"When was Nike incorporated?\",\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dd2770e-0a8e-429d-9f83-50f74bf56543",
   "metadata": {},
   "source": [
    "`VectorStoreRetriever` 支持三种搜索类型：  \n",
    "- `\"similarity\"`（相似性），这是默认的搜索类型，  \n",
    "- `\"mmr\"`（最大边缘相关性，用于在结果中平衡与查询的相关性和结果之间的多样性），  \n",
    "- 以及 `\"similarity_score_threshold\"`（相似性得分阈值）。  \n",
    "\n",
    "我们可以使用最后一种搜索类型（`\"similarity_score_threshold\"`）来根据文档的相似性得分设定一个阈值，只有得分超过该阈值的文档才会被检索器输出。\n",
    "\n",
    "检索器可以轻松地集成到更复杂的应用中，例如**检索增强生成**（RAG）应用。  \n",
    "这种应用会将用户提出的问题与从向量库中检索到的相关上下文结合起来，形成一个完整的提示词（prompt），然后输入给大型语言模型（LLM）进行回答。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03615472-4318-4150-8d03-ff0f3abcba85",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
